Flow state aware management of qos through dynamic aggregate bandwidth adjustments

ABSTRACT

Conventional packet network nodes react to congestion in the packet network by dropping packets in a manner which is perceived by users to be indiscriminate. In embodiments of the invention, indiscriminate packet discards are prevented by causing packets to be discarded on lower priority flows and flow aggregates. Further action is taken to reduce the likelihood of packet discards. When an aggregate set of flows raises a congestion alarm, action is taken to try to increase aggregate capacity by excising capacity from pre-assigned donor aggregates. A donor aggregate may be carrying flows, for example, classified as best effort. Another type of donor capacity is donor re-assignable unused capacity. Aggregates may have capacity added either up to a defined limit or, temporarily, exceeding any limit provided there is free capacity available, but removable back to the defined limit when other aggregates need increased capacity and are below their defined limits.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. application Ser. No.15/688,712, filed on Aug. 28, 2017, which is a continuation of U.S.application Ser. No. 14/171,605, filed on Feb. 3, 2014, which is acontinuation of U.S. application Ser. No. 12/828,150, filed on Jun. 30,2010, which claims priority to U.S. Provisional Application No.61/221,830, filed on Jun. 30, 2009, which are all hereby incorporatedherein by reference as if set forth in full.

BACKGROUND Field of the Invention

The present invention relates to a communications network and a methodof operating a communications network.

Description of the Related Art

Recently, the demand for streaming video to a computer via the Internethas grown strongly. This has led to a need to supply increasing amountsof video material over local communication networks (including thecopper pairs used by telephone network operators or the coaxial cablesused by cable television network operators).

In telephony networks this additional demand is being met partly by theintroduction of Digital Subscriber Loop (DSL) technology. As its namesuggests, this technology carries digital signals over the local copperloop between a user's home and a local telephone exchange. Data-rates ofseveral megabits per second to the user's home have become commonplace.Advances in this technology now lead to much higher rates, 20 Mbit/s andabove and with plans in the industry to offer 50 Mbit/s and above to asubstantial proportion of broadband end users in the relatively nearfuture. Using DSL, the digital signal is conveyed between modems placedat either end of the copper loop. The advantages of statisticalmultiplexing have led to the digital signals being organized intotransport packets (whether they be Asynchronous Transfer Mode (ATM)packets or Ethernet packets. Over either of these are carried InternetProtocol (IP) packets conveying all of the broadband services.

Cable networks have also been upgraded to carry broadband services touser's homes. Substantial numbers of users currently receive broadbandservices over cable and, again, the services are conveyed using IPpackets over Ethernet.

Video material requires a data rate which varies between 1.5 Mbps (for aquality comparable to that offered by a video cassette recording) to 20Mbps (High-Definition Television). In DSL or cable networks, thesehigher application data rates means that a mechanism is needed to managecontention for the capacity available towards the user's home. Thiscapacity management includes both the maximum available capacity for anyone user and the shared capacity towards the DSL Access Multiplexer(DSLAM) or Cable Modem Termination System (CMTS). This capacity(sometimes termed “backhaul” capacity) may be shared by several hundredusers, and contention for the capacity will also need to be managed asusers demand more choice in the material they view.

The backhaul capacity may be divided into VLANs, where each VLAN is theaggregate bandwidth that a single internet service provider can exploitto deliver services to many end-users. In this case, the active userswho are receiving internet services via this service provider will sharethe capacity of this VLAN aggregate. In an alternative arrangement,several internet service providers may jointly share the backhaulcapacity and some means may be provided so that each service providerobtains a fair share.

Under the first arrangement, where the capacity of a VLAN is availableto a single service provider, there is freedom for that service providerto make unique choices about cost and QoS. The choice is a balancebetween insufficient capacity would result in frequent congestion andmore dissatisfaction among the end users. Too much capacity results inexcess cost for the provision of services.

Under the second arrangement, where several service providers sharecapacity, there may be more opportunity to exploit unused capacity thatbelongs to one of the fair shares. This has advantages but also somedisadvantages. The advantage is that the cost of aggregate bandwidth maybe kept lower for each of the service providers. Instead they may relyon the availability of unused bandwidth when otherwise there would havebeen dissatisfaction among their end users due to congestion. Thedisadvantage is that the availability of extra bandwidth is uncertain,being dependent on the patterns of usage of end users of each of thesharing service providers. Another disadvantage is the non-uniqueness ofthe cost/QoS offering of each of the service providers. However, thisdisadvantage can be overcome by altering the nature of the fair-sharecontrols of the aggregate bandwidth, so that one service provider ispreferred over another in terms of the amount of capacity its users areentitled to get when loading levels are high.

An arrangement where capacity is shared among several service providersand the capacity is unequally divided among them when loading levels arehigh can lead to even greater uncertainty that extra capacity is therewhen needed, especially for those service providers that receive a loweramount of capacity that the average.

It is the management of this issue to remove the uncertainty that extrabandwidth is there when needed and to provide just enough bandwidth, onbackhaul links or other network links, that is the subject of thepresent patent.

In a conventional circuit-switched telephone network, the problem ofcontention for scarce telecommunication resources is dealt with bysimply preventing a user from receiving (or sending) any traffic unlessthe necessary capacity to carry that traffic can be reserved beforehand.The capabilities of multi-services packet networks have also beendeveloped to include call admission control schemes—examples include theResource Reservation Protocol (RSVP). Although such schemes can preventcongestion when all new communications or calls are admitted or rejectedusing these principles, QoS management must also manage so-called“elastic” traffic where there is potentially a need for a minimumguaranteed rate but frequently a desire to transmit the flow as fast aspossible, subject to network congestion constraints and constraints onmaximum sending rates.

An alternative to the use of connection admission control in packetnetworks is to use reactive flow control. These schemes allow usersaccess to communications resources but attempt to cause senders todecrease their sending rate on the onset of congestion. The scheme usedfor reliable transmission across the Internet (Transmission ControlProtocol) is the most common example. This is unsuitable for video flowshowever, since real-time video servers cannot reduce their sending rate.

Most flow control schemes, applied to elastic applications, result insome reduction of the rate available at the onset of congestion. Someflow control schemes are more sophisticated, classifying traffic intodifferent classes, with some classes being more likely to suffer packetdelay or discard than others. In situations where such classification isnot available or where most traffic is within one class, alternativesolutions must be provided. One such alternative solution whichconcentrates the adverse effects of ATM cell discard on one IP packet ata time is described in ‘Early Selective Packet Discard for AlternatingResource Access of TCP over ATM-UBR’ by Kangsik Cheon and Shivendra S.Panwar, in the Proceedings of IEEE Conference on Local Computer NetworksLCN 97, Minneapolis, Minn., Nov. 2-5, 1997, incorporated by referenceherein.

A discussion toward the Internet based Next Generation Network (NGN) isactively progressed around the standards bodies including the ITU-T(International Telecommunication Union - TelecommunicationStandardization Sector) and ETSI (European Telecommunications StandardsInstitute), IEEE (Institute of Electrical and Electronics Engineers),IETF (Internet Engineering Task Force), and etc. The roles of thestandard bodies are different. The IEEE and IETF develop the coretechnology for specific problems in layer 2 and layer 3, respectively.ITU-T and ETSI develop the network architecture and control procedure.

A QoS control or resource control architecture has been developed in theseveral standard bodies. To name a few, they are ITU-T, ETSI, Cable Lab,3GPP, MSF, and the DSL forum. Among those organization, CableLab, andDSL forum, 3GPP, and ETSI define the QoS control architecture in aparticular case while ITU-T defines the generic architecture that cancover the outcomes of other standard bodies.

CableLab defines the dynamic QoS (DQoS) control architecture asdescribed in PacketCable Specification “PacketCable DynamicQuality-of-Service” for the Hybrid Fiber and Coaxial (HFC) network. Thecontrol architecture is designed for the uniqueness of the HFC network.In the HFC network, multiple CMs (Cable Modems) share an upstreamchannel to CMTS (Cable Modem Termination System). The bandwidth iscontrolled based on layer 2 MAC protocol called DOCSIS (Data Over CableSystem Interface Specification) as described in “Data-Over-Cable ServiceInterface Specifications, Radio Frequency Interface Specification”, byCable Television Laboratories, Inc., Jul. 30, 2003. The layer 2 levelQoS guarantee mechanism is defined from the DOCSIS version 1.1. The goalof the DQoS is supporting the QoS guaranteed service through HFCnetwork.

DQoS defines the procedure of the call setup signaling and the dynamicQoS control on DOCSIS interface. In the architecture, the CMS (CallManagement Server) I Gate controller controls the call establishment.The guaranteed bandwidth between CM and CMTS is reserved dynamicallyduring the call setup signaling. The CMS/Gate Controller triggers thelayer 2 or layer 3 QoS signaling to reserve the bandwidth in the HFCnetwork by sending commands to CM, CMS, or MTA (Multimedia TerminalAdapter).

DQoS has been refined through version 1.0, 1.5, and 2.0. Version 1.0defines the basic call setup signaling procedure for both embedded MTAand standalone MTA. The embedded MTA can initiate the dynamic layer 2QoS signaling while a standalone MTA initiates IP level QoS signaling.Version 1.5 and 2.0 defines the QoS control architecture when SIP(Session Initiation Protocol) based call setup signaling is used. DQoS2.0 is defined especially for interoperability with IP MultimediaSubsystem (IMS) which is the SIP based call setup architecture developedin 3^(rd) Generation Partnership Project (3GPP). PacketCable Multimedia,as described in PacketCable Specification “Multimedia Specification”,Dec. 21, 2005, has been developed for simple and reliable control forthe multimedia service over cable network. It defines the servicedelivery framework for the policy based control on multimedia service.The simple procedure for time or volume based resource authorization,resource auditing mechanism, and security of the infrastructure aredefined in PacketCable Multimedia.

Such developments as this strongly suggest that new QoS mechanismsshould take account of, and build on top of, the underlying deploymentof QoS controls.

Again, the DSL forum defines the resource control at the DSL (DigitalSubscriber Line) access network, as described in Technical Report 59 DSLForum “DSL Evolution—Architecture Requirements for the Support ofQoS-Enabled IP Services”. Unlike Cable network, DSL modem is connectedto the subscriber through the dedicated line. Layer 2 level dynamic QoScontrol between DSL modem and Digital Subscriber Line Access Multiplexer(DSLAM) is not required. The DSL forum focuses more on resource controlin the home network especially resource control of multiple terminalsbehind the home gateway.

The resource control architectures defined in the above mentioned twostandard bodies—PacketCable and DSL Forum focus on a specific transporttechnology (i.e., HFC network and DSL network). The scope of DQoS andDSL forum is mainly within network operator's view. Unlike these, RACF(Resource and Admission Control Functions), as described in ITU-Trecommendation Y.2111 “Resource and Admission Control Functions in NGN”,and RAGS (Resource and Admission Control Sub-system), as described inETSI ES 282 003 V1.1.1 (2006 March), “Resource and Admission ControlSub-system (RAGS); Functional Architecture”, define the resource controlarchitecture in more general aspect.

The QoS control architecture in both RACF and RAGS are closely relatedwith 3GPP (3rd Generation Partnership Project) effort. The 3GPP isoriginally founded for developing new service architecture over cellularnetwork, especially for GSM (Global System for Mobile communication)network. During this effort, 3GPP developed the IMS (IP MultimediaSubsystem) for controlling the IP multimedia services in the areas ofsession control, service control, and management of database of thesubscribers. Even though IMS is initially developed for the evolution ofGSM cellular network, its framework can be applicable for any types oftransport technologies. The IMS architecture has been adopted to theother QoS control architectures such as 3GPP2 MMD (Multimedia Domain),ETSI TISPAN (Telecoms & Internet converged Services & Protocols forAdvanced Networks), and ITU-T NGN. Thus, both RAGS and RACF areinteroperable with IMS.

In general, RACF and RAGS are very similar with each other. The twostandards bodies are closely interacted in developing theirarchitecture. There is no significant conflict between the two, butthere are still differences, as described in ITU-T NGN-GSI Contribution,“Comparison of TISPAN RAGS and ITU-T RACF”. One of differences is therange of the control region. The control region of RAGS covers theaccess network and the edge of the core network. The access network isdefined as the region where the traffic is aggregated or distributedwithout dynamic routing. The resource control in the access network isdone in layer 2 level. The core network is the region that the IProuting starts. The core network is out of scope in the RAGS. RACF,however, covers both core and access network. RACF covers both fixed andmobile networks while RAGS is defined for the fixed network. For thecontrol mechanism, the RACF defines more control scenarios than RAGS.Therefore, RAGS is considered as a subset of RACF.

ITU-T defines QoS control functions based on its NGN architecture. Oneof the important concepts in the ITU-T NGN architecture is theindependence of the transport and the service, as described in ITU-Trecommendation Y.2012 “Functional Requirements and Architecture of theNGN”. The transport is concerning about the delivery of packets of anykind generically, while the services are concerns about the packetpayloads, which may be part of the user, control, or management plane.In this design principle, the NGN architecture is divided into twostratums—Service Stratum and Transport Stratum. Under the concept of theindependence of a service and transport functions, the network resourceand reliability are guaranteed by the network side upon request from theservice stratum. Service Stratum is responsible for the applicationsignaling and Transport Stratum is responsible for reliable data packetforwarding and traffic control. The service stratum can be a simpleapplication server or a full-blown system such as IMS (IP MultimediaSub-system).

Transport control function is located in Transport stratum interfacingwith the Service stratum. It determines the admission of the requestedservice based on the network policy and the resource availability. Italso controls the network element to allocate the resource once it isaccepted. Resource and Admission Control Functions (RACF) is responsiblefor the major part of the admission decision and resource control of thetransport function. Details of RACF mechanism can be found in “Overviewof ITU-T NGN QoS control”, by Jongtae Song, Mi Young Chang, Soon SeokLee, and Jinoo Joung, IEEE Communication Magazine, Vol. 45, No. 9,September 2007 and ITU-T recommendation Y.2111 “Resource and AdmissionControl Functions in NGN”, incorporated by reference herein.

This developing infrastructure needs to be taken account of whenconsidering new QoS mechanisms.

Review of current per-flow QoS controls Flow level transport technologyis not a new concept. The core technologies for traffic managementschemes such as flow level scheduling, policing, and sharing are alreadyavailable in a commercial product, as described in “Flow based controlfor Future Internet” by Jongtae Song, presented in Future Internet Forum(FIW) in July 2007, incorporated by reference herein. The currentdeployment of flow base control, however, is limited only at the edge ofthe network. Typical examples of flow base control are trafficmonitoring and packet inspection, PacketCable access, session bordercontroller, edge router, and interworking between two networks. They aremostly stand-alone solution at the edge of the network.

However, flow level traffic control only at the edge cannot guaranteethe flow level QoS. Furthermore, DiffSery guarantees the QoS only if thepremium traffic load is very low (˜under 10%), as described in“Providing guaranteed services without per flow management” by I. Stoikaand H. Zhang in CM SIGCOMM, September 1999, pp. 91-94, incorporated byreference herein. On the other hand, having scalable controlarchitecture for flow level traffic control along the data path is achallenging issue, because the number of flows in a network is huge.

There are several schemes proposed for the scalable control of trafficusing flow level mechanisms. These are listed below.

(1). Flow Aware Network (FAN) France Telecom proposed a Flow AwareNetwork (FAN), as described in “A new direction for quality of service:Flow-aware networking” by S. Oueslati and J. Roberts in Proc. Conferenceon Next Generation Internet Network (NGI), April 2005, incorporated byreference herein. FAN applies three different regimes based on thenetwork status. They are the “transparent regime”, “elastic regime”, and“overload regime”. The transparent regime is applied when the networkhas no congestion at all. The elastic regime is applied when the networkexperiences the occasional traffic congestion because of a few high ratedata flows. The overload regime is applied when the traffic overloadsthe link capacity in the network.

No traffic control is required in the transparent regime. The trafficcontrol is effective only in the overload or elastic regime. In theelastic regime, the network enforces the bandwidth limit for every flow.Every flow is assigned the same amount of bandwidth. In the overloadregime, new flows are blocked to protect existing flows. To reduce thecontrol complexity, an implicit approach is preferred where no signalingis required for controlling the network. Each node makes locally optimaldecision based on local observation.

The main focus of FAN is the simplicity. It requires no signaling. Onlyimplicit admission control is required upon congestion. Although thecontrol mechanism is very simple, it is shown that the network isstabilized remarkably in FAN. However, this architecture is designedmainly for network stabilization aspect. Every flow is treated equally.In order to support various of QoS requirement for individual flow, thisarchitecture should be improved.

(2). Flow Sate Aware (FSA) technologies FSA is developed to providedifferent QoS for the individual flow. FSA defines the service typesbased on typical example of Internet services, as described in ITU-TRecommendation Y.2121, “Requirements for the support of statefulflow-aware transport technology in an NGN” and “Changing the Internet tosupport real-time content supply from a large fraction of broadbandresidential users” by J. L. Adams, L. G. Roberts, and A. I. Jsselmuiden,BT Technology Journal, Vol. 23, No. 2, pp 217-231, April 2005,incorporated by reference herein. They are Maximum Rate (MR), GuaranteedRate (GR), Variable Rate (VR), and Available Rate (AR). GR is designedfor applications requiring guaranteed bandwidth for the entire durationof the flow. MR is designed for streaming media such as video and voice.AR is designed for data traffic flow where the application can setup theflow rate at the maximum rate that the network can currently support. VRis the combination of AR and MR. VR could be used for obtaining amaximum response time for a transaction (e.g., a stock trade withmaximum transaction time). The MR portion guarantees the minimumguaranteed bandwidth and AR portion is for use available networkresource. FSA divides the network resource into two portions. One isFixed Rate (FR) and the other is Network Rate (NR). FR is requested whenflow needs a fixed rate available during the service. NR is requestedwhen flow sends buffered data using network available bandwidth. Servicetype GR and MR request FR, AR requests NR, and VR requests both NR andFR. The detail requirement is defined in ITU-T Recommendation Y.2121,“Requirements for the support of stateful flow-aware transporttechnology in an NGN”.

FR and NR are requested by the signaling, as described in ITU-T StudyGroup 11, Draft Recommendation Q.flowstatesig on signaling protocols andprocedures relating to Flow State Aware access QoS control in an NGN,Editor J. L. Adams, incorporated by reference herein, and every nodealong the path configures its resource based on the requested FR and NR.For the call setup signaling, the source node and destination nodeexchanges the control messages. [Figure 1] describes the signalingprocedure for the service type MR, GR, and AR. In the ingress FSA (iFSA)and egress FSA (eFSA) exchanges the request, response, confirm,renegotiate, and confirm message for request the transport resource. ForMR, iFSA sends the data traffic before receiving the response from eFSA.MR is designed based on the concept of the conditional guaranteedbandwidth, as described in “Changing the internet to support real-timecontent supply from a large fraction of broadband residential users” byJ. L. Adams, L. G. Roberts, and A. I. Jsselmuiden, BT TechnologyJournal, Vol. 23, No. 2, pp 217-231, April 2005, and ITU-TRecommendation Y.1221 Amendment 2 (2005), Traffic control and congestioncontrol in IP-based networks, incorporated by reference herein. For GR,it needs to know the explicit start and ending time of the flow.Therefore, it sends confirm and close messages for acknowledging everytransit node reserves and release the requested bandwidth. The servicetype AR is designed to use network available resource. iFSA and otherFSA nodes continuously monitor the network available resource and adjustthe NR accordingly.

Both approaches, FAN and FAS, give an insight for flow based trafficcontrol. FAN shows that even very simple flow level traffic control canstabilize the network efficiently. FSA shows that the network resourcecan be divided into FR and NR. It also indicates that the transit nodesshould be controlled for end-to-end flow level QoS.

However, the two approaches have outstanding issues. As mentionedearlier, FAN is not designed for supporting various QoS requirement ofthe service. Its main objective is stabilizing the overall networkperformance. In this viewpoint, FAN treats every flow equally. This maystabilize the transport network in general, but the network providercannot generate additional profit, because FAN cannot support theservice that has special QoS treatment. Good business model is hardlyfound in this case.

FSA is designed for supporting various QoS requirement. Itsimplementation can be done in both the in-band signaling, as describedin ITU-T Study Group 11, Draft Recommendation Q.flowstatesig onsignaling protocols and procedures relating to Flow State Aware accessQoS control in an NGN, and out-of-band signaling. The in-band signalingprocedure requires the every node exchanges the request and response.The request need to be examined by the all the transit node. Thedestination node generates the response message, and source node findsthe agreed rate from the response message. In this approach every FSAnode should maintain the flow state.

Requiring FSA signaling feature in every user terminal is possible.However, by making the terminal independent of FSA, we can have severalbenefits. First, the terminal usually has different capability. Thenetwork architecture should be flexible enough to support multiple typesof terminal in a network. The terminal can support transport QoSsignaling but also has application signaling. The application signalingis common for all terminal types. In order to support more terminaltypes, the QoS signaling of terminal should be designed in applicationlevel. Second, the network security is important in managed network.Enabling the signaling function in the terminal may cause the securityhole in the network. For resolving this problem, ITU-T RecommendationY.2121, “Requirements for the support of stateful flow-aware transporttechnology in an NGN” specifies the mechanism to authorize the in-bandsignaling in the application signaling phase. FSA signaling initiated inthe network side from the network edge can be another option to avoidthe security problem.

In both FAN and FSA approaches, focus is mainly in the transportcontrol. In order to take account of the existing deployment of QoSfunctions, the concept of RACF function needs to be considered.

(3). FSA with out-band signaling. In this proposal, the FSA signaling iscombined with RACF. CPE (Customer Premises Equipment) or user terminalshould be able to request the flow level resource in any kind ofapplication signaling. In this aspect, the CPE and user terminal shouldbe protocol independent.

Second, this proposal focuses flow-based control in the access network,not the core. In the access network, user data traffic is staticallyrouted to the edge of the core network, and the downstream data trafficis statically forwarded from the edge of the core network to the enduser. Core supports both IP based dynamic routing and layer 2 basedstatic forwarding. The traffic volume, number of flows, and dynamicityof traffic are different in the core and access. Traditional accessnetwork controls the bandwidth based on subscribed bandwidth per user inL2 level. For flow level traffic control, however, the bandwidth shouldbe controlled by individual flow. Flow awareness capability is requiredin the access nodes. The static packet forwarding and scheduling in theflow level granularity is required in the access network. The call bycall traffic control and policy enforcement from control plane (e.g.,RACF) should be done in micro flow level. In the core, the number offlows is high and call by call flow level control in RACF is difficultto achieve. In the core side network, therefore, the traffic should becontrolled in aggregate level rather than micro flow level. Thereliability and monitoring capability will be more important in thecore. The flow based traffic control and the aggregated traffic controlshould be translated at the edge of the core network.

These design principles are further illustrated in FIG. 2.

However, none of the above proposals provides a method of managingcontention in a packet network which allows flow-based QoS mechanisms tooperate without end-user signaling and support:

-   -   Preference priority control of some flows in the event of        congestion or sudden re-routing of traffic in the event of a        network link failure.    -   Admission of variable rate, delay-sensitive flows requiring some        minimum guaranteed bandwidth.    -   Management of the fastest transfer time (highest available        transfer rate).

In ‘Flow State Aware QoS Management Without User Signaling’, U.S.Provisional Application No.: 61/118,964 filed Dec. 1, 2008, and ‘FlowState Aware QoS Aggregate Management Without User Signaling’, U.S.Provisional Application No.: 61/185,843 filed Jun. 10, 2009, solutionsare described that do not require signaling. Both solutions assume afixed assignment of capacity is provided to each aggregate VLAN.Solution in U.S. Provisional Application No.: 61/118,964 filed Dec. 1,2008 gives every flow either a guaranteed rate or some assignment of aminimum rate and some assignment of the remaining unused capacity whichis adjusted according to the number of flows and the preference priorityof a flow. Solution in U.S. Provisional Application No.: 61/185,843filed Jun. 10, 2009 does not assign capacity to a single flow and doesnot discard any packets of any flows until the fixed capacity of anaggregate is nearing congestion. It causes less packet deletions thanU.S. Provisional Application No.: 61/118,964 filed Dec. 1, 2008,although, if there is evolution towards some use of signaling for someof the services and for some of the flows, U.S. Provisional ApplicationNo.: 61/185,843 filed Jun. 10, 2009 is potentially more unfair on thoseflows that are being policed through signaling while other flows,established without signaling, are free to change their ratesarbitrarily. In this case, U.S. Provisional Application No.: 61/118,964filed Dec. 1, 2008 is believed to provide a fairer arrangement. Theaforesaid patent applications are incorporated by reference herein.

However, both solutions in U.S. Provisional Application No.: 61/118,964filed Dec. 1, 2008 and U.S. Provisional Application No.: 61/185,843filed Jun. 10, 2009 assume the aggregate VLAN capacities are constant.The choice of the capacity is important to the service provider. Thereis a need to avoid congestion becoming too frequent. However, there maybe uncertainty about the traffic load, especially with the take-up ofnew applications among the end users. Therefore, it is typical for theservice provider to provide excess capacity to ensure congestion isinfrequent. But, as application data rates continue to increase, thisapproach of always having excess capacity becomes more and more costlyfor the service provider and, potentially, more costly for the user.

Thus, novel methods of operating a communications network are needed.

SUMMARY OF THE INVENTION

The inventive methodology is directed to methods and systems thatsubstantially obviate one or more of the above and other problemsassociated with conventional techniques for operating a communicationsnetwork.

Various embodiments of the present invention overcomes theabove-mentioned difficulties, allowing service providers to exploitboth:

-   -   flow state aware QoS; and    -   the rapid adjustment of VLAN capacity as needed.

This achieves the significant reduction of both the duration andfrequency of incidents of congestion at minimum cost in terms ofcapacity deployed.

According to the present invention, there is provided a method ofoperating a packet subnet that aims to solve the problem of managingcontention in a packet network allowing:

-   -   Preference priority control of some flows in the event of        congestion or sudden re-routing of traffic in the event of a        network link failure.    -   Dynamic adjustments of bandwidth in the event of congestion or        sudden re-routing of traffic in the event of a network link        failure, or dynamic adjustments of bandwidth in the event of one        or more flow-based or aggregate flow-based measurements that are        indicative of a change of bandwidth on one or more aggregates to        meet QoS or cost objectives    -   These features to be realized in an environment where there may,        or may not, be FSA signaling and where RACF may or may not be        deployed.

According to a first aspect of the embodiments of the present invention,the said method of operating the subnet comprises:

-   -   The partitioning of one or more physical links into a number of        aggregates, where each aggregate is assigned an amount of        capacity and one or several aggregates are assigned to one or        more donor aggregates co-existing on the same said link or links        as the said assigned aggregates, so that any one of the said        aggregates may increase its capacity by borrowing capacity from        one or more of its said donor aggregates, or may decrease its        capacity by giving back capacity to one or more of the said        donor aggregates.    -   Receiving one or more communication packets indicative of the        commencement of a packet communication through said subnet, said        packet communication comprising a flow consisting of a plurality        of packets.    -   Responsive to recognition of the commencement of a new packet        flow, storing a flow identifier, enabling identification of        packets belonging to said commenced packet communication flow        and associating that flow identifier with one or more of the        said aggregates, including or excluding the said donor        aggregates.    -   Responsive to recognition of the commencement of said new flow,        retrieving policy information relating to any initial or        sustained QoS treatment of the flow, where such policy        information is derived from:

packet header information at any level of encapsulation;

signaling information related to that flow identity;

policy attributes associated with the physical link on which the packetarrived or will depart;

measured rates applying to the flow rate of the said flow, or themeasured jitter in flow rate; and

measured rates applying to the total traffic on one or more of the saidaggregates which the said new flow is associated with, or the measuredjitter in such said rates.

-   -   Responsive to recognition of the commencement of said new flow,        and to retrieved policy information, determining an initial or        sustained discard probability from both:    -   the associated stored policy information corresponding to the        said new flow.    -   packet counters, byte counters, or rate counters of any        aggregate group of flows to which this flow belongs that are        being monitored with respect to one or more traffic load        criteria.    -   Responsive to the said policy determination, and QoS treatment        determination, related to flow identity, retrieving said policy        determination and QoS treatment determination associated with        the flow of the latest arriving packet, and determining if one        or several pre-determined thresholds or limits have been        reached, or will be reached if the latest arriving packet is        transmitted, that provide indications of a change in the        capacity requirements of aggregates associated with the flow of        this said latest arriving packet, said pre-determined thresholds        being based on, but not limited to:

Flow rate or jitter in rate measured on any flow;

Flow rate or jitter in flow rate measured on any aggregate;

Flow count in any aggregates;

Flow discard probability on the flow of the latest arriving packet;

Policy information on any aggregates associated with the flow of thelatest arriving packet;

Packet discard measurements on any aggregate;

Packet discard measurements on any flow;

Packet delay measurements on any flow or averaged on any aggregate;

Received signaling related to any flow, or received signaling related toany aggregate.

-   -   Responsive to said determination that one or more pre-determined        thresholds or limits have been reached on a said aggregate        associated with the flow of the latest arriving packet, causing        an indication to be observable that an increase or decrease in        capacity is required for that aggregate and a corresponding        decrease or increase of capacity is required of one or more        donor aggregates associated with that aggregate.    -   Enabling the immediate transfer of the said latest arriving        packet subject only to the said determination of the discard        probability associated with the flow identity of the said        packet.

According to this first aspect of the embodiments of the presentinvention, allowing the said pre-determined limits or thresholdsindicative of a change in bandwidth requirement on one or moreaggregates associated with the flow identity of the said latest arrivingpacket to include:

-   -   Measurements of rates or jitter in rates associated with a group        of flows which may or may not include the said flow of the        latest arriving packet, and where this said group of flows is a        subset of all the active flows of one or more of the said        aggregates associated with the said flow identity of the said        latest arriving packet;    -   A preference priority associated with the flow identity of the        said latest arriving packet, derived by the said policy        determination and the said QoS treatment determination or        through the said signaling received about the said flow.

According to this first aspect of the embodiments of the presentinvention, optionally a method where the said observable indication thata change in aggregate capacity is required on one or more aggregatesincludes the sending of an indication towards a verification functionthat determines if the said indicated change in capacity can bepermitted or modified.

According to this first aspect of the embodiments of the presentinvention, optionally a method where the said verification functionreduces the capacity of a donor aggregate by sending an indicationtowards a policing function that lowers the policing thresholds on rateor rate jitter of said donor aggregate such that, when said loweredpolicing thresholds are determined to be exceeded by measurement of thelatest and recent arriving packets of the total traffic of said donoraggregate, a number of packets are discarded instead of beingtransmitted so that the remaining transmitted packets do not exceed thelowered policing rates.

According to this first aspect of the embodiments of the presentinvention, optionally a method where the said verification function,having observed said indication for an increase in capacity on anaggregate, optionally determines for this said aggregate which donoraggregates associated with the said aggregate shall be decreased incapacity and by what amount of capacity. Similarly, where theverification function observed a said indication for a decrease incapacity of the said aggregate, which donor aggregates associated withthe said aggregate shall be increased in capacity and by what amount ofcapacity.

According to this first aspect of the embodiments of the presentinvention, optionally a method where a donor aggregate has no flows andits capacity is an unused pool of capacity that can be used to increasethe capacity of other aggregates and its capacity can be increased bythe release of capacity from other aggregates.

According to this first aspect of the embodiments of the presentinvention, optionally a method where the said unused pool of capacity isshared among several other flow aggregates and the said verificationfunction controls the sharing of its pool of capacity such that:

-   -   The said verification function allows temporary use of any spare        capacity in the unused capacity pool for any aggregate        indicating that an increase in capacity is required on the basis        of temporary or unguaranteed assignments above a predetermined        share of the said pool of capacity.    -   The said verification function decreases said temporary use of        spare capacity when other aggregates require an increase in        capacity and one or more aggregates are currently using more        than a pre-determined share of the pool of capacity, such that        no aggregate is decreased in capacity below its said        pre-determined share unless it indicates that a further decrease        is required.

According to this first aspect of the embodiments of the presentinvention, optionally a method where the said unused pool of capacity isshared among several other flow aggregates and the said verificationfunction controls the sharing of its pool of capacity such that:

-   -   The said verification function allows guaranteed assignments of        additional capacity for any aggregate indicating that an        increase in capacity is required on the basis of guaranteed        assignments up to a pre-determined share of the said pool of        capacity.    -   The said verification function decreases capacity of any        aggregate requiring guaranteed assignments of capacity only        when:

Under normal operating conditions the said aggregate requiringguaranteed assignments of capacity indicates to the verificationfunction that a decrease in capacity should be effected;

Under abnormal operating conditions where current levels of guaranteedcapacity cannot be sustained.

According to this first aspect of the embodiments of the presentinvention, optionally a method where the said determination of aninitial or sustained discard probability associated with a flow identityincludes the assignment of a larger discard probability when, for anyaggregate associated with the said flow identity, being an aggregatethat has a portion of its capacity assigned on the basis of the saidtemporary assignment of capacity, the measured rate of that aggregate isgreater than one or more threshold rates based on the saidpre-determined share of capacity available to that aggregate.

According to this first aspect of the embodiments of the presentinvention, optionally a method where, when a packet arrives whose flowis associated with an aggregate where congestion has ceased or wheretraffic levels on an aggregate have fallen below some thresholdindicative of the onset of congestion, or when measurements relating toany rate-based, byte count-based, or packet-count based threshold showsthat measured levels are now below level(s) that had previously exceededone or more such thresholds, then retrieving policy relating to such achange of status of this aggregate and relating to re-grading of thediscard probability of the said flow, given that measured levels havefallen or congestion has disappeared.

According to a second aspect of the embodiments of the present inventionthere is provided a packet network node comprising:

-   -   an input for receiving one or more packets;    -   means arranged in operation to measure a packet rate or byte        rate in said packet network node, relating to any flow or group        of flows, and means to detect an excessive packet rate or byte        rate, including the use of any associated limits of rate or rate        jitter assigned to such a flow or group of flows;    -   means to re-assign capacity and any associated rate limits or        rate jitter limits between either of the following:    -   one group of flows and one or more other groups of flows,        referred to here as donor groups, that have a common route and        share capacity, or have a partition of the capacity, of a link        or a number of consecutive links over which the first said group        is routed;    -   one group of flows and a combination of additional other said        donor groups of flows (if any) and unused re-assignable        capacity, referred to here as donor re-assignable unused        capacity on a shared common route that the said first group,        said donor groups (if any) and said donor re-assignable capacity        share.    -   means arranged in operation to detect excessive packet bursts or        jitter in the packet or byte rate in said packet network node        relating to any flow or group of flows including the use of        associated limits of burstiness or jitter assigned to such a        flow or group of flows;    -   means arranged in operation to retrieve policy determination and        QoS treatment determination associated with the flow of the        latest arriving packet, and means arranged in operation to        determine if one or several pre-determined thresholds or limits        have been reached, or will be reached if the latest arriving        packet is transmitted, that provide indications of a change in        the capacity requirements of groups of flows associated with the        flow of this said latest arriving packet, said pre-determined        thresholds being based on, but not limited to:    -   Flow rate or jitter in rate measured on any flow;    -   Flow rate or jitter in flow rate measured on any group of flows;    -   Flow count in any groups of flows;    -   Flow discard probability on the flow of the latest arriving        packet;    -   Policy information on any groups of flows associated with the        flow of the latest arriving packet;    -   Packet discard measurements on any group of flows;    -   Packet discard measurements on any flow;    -   Packet delay measurements on any flow or averaged on any group        of flows;    -   Received signaling related to any flow, or received signaling        related to any group of flows.

According to this second aspect of the embodiments of the presentinvention, the packet network includes further means arranged inoperation to provide an observable indication of a requirement for acapacity reassignment to any group of flows, which is either:

-   -   an observable indication that an increase in capacity is        required for a group of flows and a corresponding decrease of        capacity is required of zero or more said donor groups and zero        or more said donor re-assignable capacity associated with that        first said group.    -   an observable indication that an decrease in capacity is        required for a group of flows and a corresponding increase of        capacity is required of zero or more said donor groups and zero        or more said donor re-assignable capacity associated with that        first said group.

According to this second aspect of the present invention, the packetnetwork includes further optional means arranged in operation to providea communication identifier store for storing a set of communicationidentifiers and associated state information.

According to this second aspect of the embodiments of the presentinvention, the packet network includes further means arranged inoperation to reduce or increase the discard probability associated witha flow and its related communication identifier from said communicationidentifier store when one or more predetermined conditions are metassociated with measurements in any aggregate to which the flow belongs.

According to this second aspect of the embodiments of the presentinvention, the packet network includes further means arranged inoperation to identify one or more flows that should be discarded whenthe packet rate, byte rate or the jitter of a rate in any group of flowsdetermine that such a deletion is required.

According to this second aspect of the embodiments of the presentinvention, the packet network includes further means arranged inoperation, on the detection that a packet rate, byte rate or the jitterof a rate in any group of flows is equal to or has exceeded apre-determined threshold, to identify arriving packets associated withone of said communication identifiers stored in said communicationidentifier store; and forward packets associated with any flowidentifier unless the discard probability associated with that flowidentifier is set so that all packets are to be temporarily discarded.

According to this second aspect of the embodiments of the presentinvention, the packet network includes further means arranged inoperation to reassign for any flow or group of flows the preferencepriorities or discard probability based on the result of one or moremeasurements and pre-determined thresholds related to any flow or groupof flows, including means to revise the policed rate(s) of any flow orgroup of flows, said pre-determined thresholds being based on, but notlimited to:

-   -   Flow rate or jitter in rate measured on any flow;    -   Flow rate or jitter in flow rate measured on any group of flows;    -   Flow count in any groups of flows;    -   Flow discard probability on the flow of the latest arriving        packet;    -   Policy information on any groups of flows associated with the        flow of the latest arriving packet;    -   Packet discard measurements on any group of flows;    -   Packet discard measurements on any flow;    -   Packet delay measurements on any flow or averaged on any group        of flows;    -   Received signaling related to any flow, or received signaling        related to any group of flows.

According to this second aspect of the present invention the packetnetwork includes:

-   -   Means to determine the choice of flows that should be discarded        among flows whose discard priority is the highest among the        remaining flows not already selected for discard.    -   Means to limit the flows that should be discarded based on a set        of threshold values associated with any group of flows against        which is compared measured value(s) of packet rate, byte rate or        the jitter in a rate as applied to the aggregate traffic in that        said group of flows.    -   Counting means to determine how many flows have already been        selected for discard in that said group of flows.    -   Means to automatically select for discard the latest flow that        has started with a given discard probability.    -   Means to automatically select a group of flows for discard where        the group of flows all have a given discard probability.    -   Means to de-select a flow that had been selected for discard        when the said threshold values are compared against the said        measured value(s) of a said group of flows or of a flow,        together with means to determine that the latest measured        value(s) exceed only a lower threshold than when the comparison        was last performed prior to this latest said comparison.    -   Means to de-select a flow for discard based upon a time interval        having elapsed during which time it had been selected for        discard.

According to a third aspect of the embodiments of the present invention,a packet network node including a packet communications sourcecomprising:

-   -   means arranged in operation to generate signaling packets after        the determination that a change in capacity should be requested        on any aggregate.    -   means to generate information and send a signaling packet        according to:

Means that indicate any change of capacity associated with anyaggregate;

Means that indicate changes to policing rates or measurement thresholdsrelating to any flow or any aggregate.

Using the said methods and means, a packet subnet operator is able totemporarily concentrate the adverse effects of congestion on selectedcommunications, whilst improving, where possible, the capacity availableto the congested aggregate, with the following advantages:

-   -   the quality of service afforded to a communication increases in        inverse proportion to its discard probability.    -   flows with high discard probability are supported with minimum        disturbance, especially when additional capacity can be added        when required.

This in turn has the advantage of being less annoying to users receivingcommunications because flows are not rejected when traffic conditionsdetermine that new flows cannot be supported with a low discardprobability and flows with a high discard probability are managed tominimize any disturbance in the received QoS.

Additional aspects related to the invention will be set forth in part inthe description which follows, and in part will be obvious from thedescription, or may be learned by practice of the invention. Aspects ofthe invention may be realized and attained by means of the elements andcombinations of various elements and aspects particularly pointed out inthe following detailed description and the appended claims.

It is to be understood that both the foregoing and the followingdescriptions are exemplary and explanatory only and are not intended tolimit the claimed invention or application thereof in any mannerwhatsoever.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification exemplify the embodiments of the presentinvention and, together with the description, serve to explain andillustrate principles of the inventive technique. Specifically:

FIG. 1 illustrates the conventional configuration relating to in-bandonly FSA signaling.

FIG. 2 illustrates alternative conventional configuration relating to acombination of in-band FSA signaling and out-of-band signaling.

FIG. 3 illustrates the high-level network configuration for therealization of flow-based services

FIG. 4 illustrates an example deployment of a dynamic aggregatebandwidth manager and FSA QoS manager, labeled in the figure as function6.

FIG. 5 illustrates an expansion of the function 6 supporting flow stateaware QoS management, measurements, aggregate alerts and alarms, anddynamic adjustment of aggregate bandwidth.

FIG. 6 illustrates an expansion of the functions of the Discard Controland Packet

Forwarding function 6.2 according to one embodiment of the presentinvention.

FIG. 7 illustrates an expansion of the functions of the Flow andAggregate Measurements, and Change Verification function 6.3 accordingto one embodiment of the present invention.

FIG. 8 illustrates another embodiment, including a signal generatorfunction 6.5 that can send signals to a remote Change Verificationfunction 6.3.1 or receive signals from such a function 6.3.1 and forwardsuch signals to a co-located function 6.3 for management of aggregaterate changes.

FIG. 9 illustrates another embodiment where arriving signaling packetsare recognized by the classifier function 6.1 and are passed to function6.5.

FIG. 10 illustrates an exemplary embodiment of a hardware platform uponwhich the present invention may be implemented.

DETAILED DESCRIPTION

In the following detailed description, reference will be made to theaccompanying drawing(s), in which identical functional elements aredesignated with like numerals. The aforementioned accompanying drawingsshow by way of illustration, and not by way of limitation, specificembodiments and implementations consistent with principles of thepresent invention. These implementations are described in sufficientdetail to enable those skilled in the art to practice the invention andit is to be understood that other implementations may be utilized andthat structural changes and/or substitutions of various elements may bemade without departing from the scope and spirit of present invention.The following detailed description is, therefore, not to be construed ina limited sense. Additionally, the various embodiments of the inventionas described may be implemented in the form of a software running on ageneral purpose computer, in the form of a specialized hardware, orcombination of software and hardware.

An internetwork (FIG. 3) comprises a user's home network 3, whereascomponent 2 of FIG. 3 is either an IP routed network or an ATM networkor an Ethernet network or layer 1 cross-connection, such as SDH. In FIG.3 is also a pair of copper wires 19 connecting the user's home network 3to network 2, an Internet Service Provider's (ISP's) local area network4, a flow-based QoS manager 6, a network link 5 that represents capacitythat the ISP uses to deliver multiple flows to (or receive from)multiple end user networks 3. The ISP network link 7 would represent theinput packets into the flow-based QoS manager 6 for flows travellingdownstream from the ISP's network 4. The ISP's local area network 4 isconnected to the Internet 8 via an Internet link 9.

The network 2 comprises exchange-housed equipment (17, 18) housed in thelocal telephone exchange building 20 and a wide-area network 22 whichconnects a plurality of such DSLAMs 18 (there is normally one or moreDSLAMs per exchange building, only one exchange building is shown in thedrawing) towards the flow-based QoS manager 6. As will be understood bythose skilled in the art, the exchange-housed equipment includes aDigital Subscriber Line Access Multiplexer (DSLAM) 18 shared betweenmany users and, for each pair of copper wires 19, a splitter unit 17which terminates the pairs of copper wires 19. The splitter unit 17 iseffective to send signals within the frequency range used for normaltelephony to the Public Switched Telephone Network (not shown) and tosend signals in higher frequency bands to the DSLAM 18.

The user's home network may comprise a PC 10, a digital television 15, asplitter unit 14, a router I set-top box 16 which incorporates anAsymmetric Digital Subscriber Line (ADSL) modem 13, a cable 12interconnecting the modem 13 and the splitter unit 14, and cablesconnecting the router 16 to the PC 10 and the digital television 15. Thesplitter unit 14 is effective to send signals within the frequency rangeused for normal telephony to the user's telephone 11 and to send signalsin higher frequency bands to the ADSL modem 13. The ADSL modem 13represents the network termination point of network 2.

The ISP's network 4 comprises an IP router 24, a content provider'svideo server 27, and a Local Area Network 30 which interconnects them.The previously mentioned Internet link 9 is connected to the IP router24.

Broadband services typically utilize Ethernet as the layer 2 protocol.Again, typically in such a case, the DSLAM terminates the ATM if PPoA isused over the copper pair, and the DSLAM encapsulates the PPP payload inEthernet for forwarding to the function 6. Link 5 is the aggregatecapacity between function 6 and the DSLAMs and it is further subdividedinto separate VLANs with either dedicated capacity per DSLAM or capacitythat can be borrowed but is pre-emptively available when trafficconditions require the full capacity that can be available towards orfrom any one DSLAM.

Two other connection products 5 are based on the IP layer and routingacross an MPLS platform.

In these cases, the PPPoE connections will terminate on a BRAS functionwithin network 22. The two options are:

-   -   Using L2TP between the BRAS (acting as the LAC) and the ISP LNS        router or direct to the service provider for them to provide the        HG functions L2TP LNS (L2TP pass through) etc.    -   Create per service provider IP VPN's.

FIG. 4 shows an example deployment of function 6 where it is used tomanage the flows that are grouped into separate VLANs, for example 40and 41. Each VLAN delivers packets towards different DSLAMS 18 and on,from there, to broadband end-users 16. Typically, each VLAN may havedynamically-controlled rate-limits which the aggregate traffic must notexceed and these limits may include rate jitter limits that occur due topacket clustering.

In FIG. 4 there are no switching functions to be performed. In thisconfiguration, the function 6 could be owned and deployed by a ServiceProvider, acting as an edge FSA shaper function and aggregate bandwidthmanager for VLANs that the Service Providers uses and exclusively owns,where such VLANs share a common route. In the example of FIG. 4 thefunction 6, if owned by a Service Provider, supports service deliveryover an Ethernet VLAN-structured fibre bandwidth connection from theNetwork Provider, enabling the Service Provider to maintain their ownFSA QoS services and Intelligent Packet services towards or from a DSLAM18 (or base station) and, ultimately via copper (or wireless) to anumber of end-users. Alternatively, in this configuration of FIG. 4, thefunction 6 is a Network Provider function enabling QoS services andIntelligent Packet Services that can be offered to Service Providers.

Function 6 shapes the aggregate traffic of a specific VLAN according tothe given maximum current rate allowable, and given current bursttolerance allowable. It also shapes the aggregate traffic towards eachend user, again according to the current maximum rate allowable andgiven current burst tolerance allowable. In each case, it applies FSAQoS principles to incidents of congestion or incidents of excessiveburstiness with respect to the individual flows that make up anyparticular aggregate traffic.

FIG. 5 relates to the first preferred embodiment and shows an expansionof the function 6, containing sub-function 6.1 to 6.4. The classifierand flow status function 6.1 uses packet header information to generatea flow identity and uses this plus additional information, including aVLAN value, or a measured rate, to look up the QoS treatment of the flowand whether the flow has just started or whether the packet is acontinuation of a previously-started flow. The results of this look upare recorded as components of the flow status of the flow and are storedin function 6.4.

The Discard Control and Packet Forwarding function 6.2 is the main QoSlogic which receives inputs from 6.1 and the Flow and AggregateMeasurements and Change Verification function 6.3. Using these inputs,function 6.2 operates to determine if the latest arriving packet shouldbe deleted or forwarded.

According to one embodiment, the method which function 6.3 uses toassist function 6.2 to determine a decision on whether to forward apacket or deleting it are:

-   -   Flows may vary in rate without any discard actions unless some        aggregate, to which the flow belongs, is experiencing some        congestion.    -   Congestion is determined by function 6.3 to be the excess jitter        in the arrival rate of the aggregate byte flow on any one        aggregate that is being monitored for such jitter conditions.    -   Function 6.3 may use different thresholds for each aggregate to        compare against measured values of the excess jitter and may use        the results of this comparison to do the following:

1. To indicate to function 6.2 different levels of congestion that havebeen reached on an aggregate, so that function 6.2 can choose theappropriate number of flows for packet discard actions and, for exampleselect the lowest priority flows first.

2. At any level of congestion, or at a threshold level below the firstlevel that would cause a congestion indication to be sent to function6.2, the function 6.3 determines whether an increase in bandwidth can beobtained on the aggregate. Or, where congestion levels are falling, orhas reached a lowest threshold level below any level that would causefunction 6.2 to apply any packet deletion to any arriving packet of thataggregate, to determine whether a decrease in bandwidth is appropriatefor the aggregate.

When such a congestion condition is notified by function 6.3.1 (see FIG.7) towards function 6.2.2 (see FIG. 6), some flows are chosen forimmediate discard.

-   -   In one embodiment, the choice of flows may be random from the        lowest priority flows but with a pre-determined limit on the        number of flows that can be chosen.    -   In another, preferred, embodiment, the initial choice of flows        includes the last flow(s) that started that are low priority.

This initial group of flows is supplemented by further flows selectedfor discard if function 6.3.1 indicates to function 6.2.2 thatcongestion has risen to a greater level on an aggregate, as determinedby comparison with the said congestion thresholds for that aggregate.

In one embodiment, when any level of congestion is detected by function6.3.3, function 6.3.1 reduces the capacity available to the entireaggregate of flows that are contained in the congested aggregate and areclassified as “best effort” flows. For example, aggregate 41 towardsDSLAM2 in FIG. 4 is contained within aggregate 40 going towards DSLAM2.In this case the aggregate 41 going towards DSLAM2 illustrates a besteffort aggregate and would be reduced in capacity if aggregate 40 goingtowards DSLAM2 became congested. This implies that the aggregate rateand rate jitter of the best effort aggregate will be policed against areduced available rate and 6.3.1 will send indications to 6.2.2 whencongestion is detected on the best effort aggregate.

In a preferred embodiment, when function 6.2.2 receives an indicationfrom 6.3.1 that a best effort aggregate is congested, it marks theentire best effort aggregate as subject to temporary discard of allarriving best effort packets until function 6.3.1 further indicates tofunction 6.2.2 that congestion has ceased on the said congested besteffort aggregate.

The function 6.2.1 (see FIG. 6) then begins to delete packets whichbelong to the flows or flow aggregates that have been selected. Thisdeletion process continues until function 6.3.1 indicates thatcongestion is diminishing and that lower thresholds have been reached,or that congestion has ceased. When a lower threshold is reached,function 6.2.2 removes some of the flows designated for discard.

-   -   In one embodiment the choice of flows that are de-selected for        discard is random among the flows that are designated for        discard.    -   In another embodiment the choice of flows is based on the        priority of the flow, with the highest priority being the most        preferred for de-selection.    -   In another embodiment, the choice of flows is based on the time        of starting of flows, with the oldest flow being the most        preferred for de-selection.

In every case above, one preferred embodiment is that sufficient flowsare de-selected until a pre-determined number of flows still remain asselected, according to the congestion level. Finally, when congestiondisappears, as indicated by function 6.3.1 towards 6.2.2, all flows arede-selected.

In a preferred embodiment, when congestion is decreasing on anaggregate, function 6.3.1 will not release any part of the bandwidththat has been assigned to the aggregate until a lowest threshold levelhas been reached on measured rate or rate jitter for that aggregate,where this lowest level is below any threshold that would cause function6.2.1 to be deleting packets of some flows or flow aggregates.

FIG. 6 is an expansion of function 6.2, showing the packet transmissionfunction 6.2.1, the main logic 6.2.2, and a packet buffer 6.2.3. As willbe understood by those skilled in the art buffer 6.2.3 operates as astore for packets which are to be sent along connection 5. The buffercomprises an area within an electronic memory. The size of the buffer6.2.3 is based on the following considerations:

-   -   In one embodiment the buffer may be removed and the operation        becomes buffer-less. The output packet rate is determined from        the input rate less any packets that are deleted, according to        any rate and rate jitter requirements of aggregate flows.    -   In another embodiment, the buffer is required to manage the        logic processing more effectively. Several packets may be        admitted to the buffer and deletion and transmission actions        performed on all of them at an appropriate point in the process        cycle time.

The functions of the main logic 6.2.2 are to receive indications fromfunction 6.3.1 and perform flow selection actions for packet discards tobe performed as already described. Similarly the deletion actions of6.2.1, responsive to the flow selection actions of 6.2.2, have alreadybeen described above. If packets are not deleted they are transmitted byfunction 6.2.1 from the buffer 6.2.3 to the output link 5.

FIG. 7 is an expansion of function 6.3, showing the main logic function6.3.1, the flow and aggregate flow rate measurements and rate alertsfunction 6.3.2, and the aggregate rate jitter measurements and alarmmanagement 6.3.3. The function 6.3.3 has been partly described above, interms of its actions detecting congestion and informing function 6.2(via function 6.3.1) of congestion alarms and state changes in thesealarms. To determine these alarms, function 6.3.3 operates as follows:

-   -   In a preferred embodiment, when a packet arrives, its byte count        is indicated by function 6.1 to 6.3.3. An amount of tokens equal        to this byte count are then placed in a set of “token buckets”        (one bucket per aggregate, choosing aggregates which this flow        belongs to, where the flow id is determined from the arriving        packet). Each token bucket is drained of tokens at a certain        pre-defined constant rate that is equivalent to the capacity        available to that given aggregate. Alarms correspond to the        depth of the bucket (i.e. the quantity of tokens in the bucket,        not yet drained away, compared to a set of threshold values.        When the quantity of tokens equals or exceeds a threshold value,        an alarm is raised. When the quantity of tokens subsequently        reduces below that same threshold, the alarm-level is reduced to        the next lower level (if any) or the alarm is ceased (if there        are no lower levels).    -   In another embodiment, the token bucket may be replaced by a        more approximate method that is simpler to calculate. A rate is        measured over a short interval determined by a starting point        when a packet arrives and a finishing point when another packet        arrives, both belonging to the same aggregate. The number of        packets used to define the interval length may be chosen at        will, with care used to ensure that it is not too long so that        very short term rate variations will be missed, and not too        short so that local clusters of packets, of acceptable size, are        not mistaken for an unacceptable local rate. Thresholds are used        to indicate alarms based on the local measured rate equaling or        exceeding one of several pre-determined thresholds.

The function 6.3.2 measures rates on individual flows and on aggregates.For aggregates, it also compares the measured rate against one or morepredetermined thresholds and indicates an alert to function 6.1 when theaggregate rate is equal to or exceeding one of these thresholds. It alsomeasures the flow rate of a flow. In a preferred embodiment it willinclude an initial measurement of the flow rate for a sufficiently shortinterval of time that allows the classifier to be quickly updated aboutthis information and potentially change the classification given to thatflow. Again, care must be used so that the interval is sufficientlyshort, yet long enough that short-term variations in the local rate, dueto packet clustering, are not confused with the average rate of theflow.

The main logic function 6.3.1 acts to receive indications from otherfunctions 6.1 and 6.2 and to schedule tasks in functions 6.3.2 and 6.3.3and to receive the outputs of these tasks and format them into outputindications towards 6.1 and 6.2.

Function 6.3.1 also includes a sub-function termed the VerificationFunction 6.3.1.2 (see FIG. 8). In another embodiment, function 6.3.1.2is located remotely operating as an Application Program running on aServer and interacting with other components of function 6.3 through anApplication Program Interface (see FIG. 9).

Function 6.3.1.1 is informed by 6.3.3 that congestion thresholds havebeen reached on an aggregate, either because congestion is increasing ordecreasing. The function 6.3.1.1 sends an indication to either aco-located function 6.3.1.2 (FIG. 8) or a remote function 6.3.1.2 (FIG.9) informing it of the aggregate identity and indicating the thresholdcongestion level. The function 6.3.1.2 retrieves policy informationrelated to the aggregate and the donor aggregates that may be used toadd more bandwidth to the congested aggregate or give back bandwidthfrom the aggregate when congestion has lowered sufficiently.

Policy decisions on the addition and subtraction of bandwidth may bevaried to suit customer needs including, but not limited to, thefollowing:

-   -   Bandwidth is not subtracted from an aggregate until the measured        bit rate of the traffic of that aggregate has fallen to a given        threshold.    -   Bandwidth is not subtracted from an aggregate until there have        been no further congestion notifications from that aggregate for        a pre-defined interval of time.    -   Bandwidth is never subtracted from some aggregates, bandwidth        additions being permanent additions.    -   Bandwidth that is subtracted from an aggregate may be all or        part of a measurement of added bandwidth that the aggregate has        received over a specified interval of time.    -   Subtracted bandwidth is re-assigned to donor aggregates in a        priority order including, but not limited to the following:

Giving bandwidth back to the donor re-assignable unused capacity onlyafter all other donor aggregates have received back specified amounts ofbandwidth.

Giving back specified amounts of bandwidth in any pre-assigned sequencethat includes all donor aggregates and the donor re-assignable unusedbandwidth, whether or not a donor aggregate or the donor re-assignableunused bandwidth gave bandwidth to the aggregate.

-   -   Subtracted bandwidth is assigned randomly to any one or more        donor aggregates and the donor re-assignable unused bandwidth.    -   Added bandwidth is assigned using a combination of donor        aggregates and the donor re-assignable unused bandwidth in a        preferred and specified sequence order, subject to defined        limits for the amount of capacity that can be added set for each        such donor and specific to the requesting aggregate.

Defined limits for the amount of capacity that can be added from a donoraggregate may be calculated based on either static unchanging values, ordynamic values based on the current traffic that the donor is carryingor the frequency which the donor is experiencing congestion.

Capacity may be added on a temporary basis above limits set for thataggregate on the basis that such added temporary capacity will beremoved when other aggregates require it where such aggregates are belowthe limits set for capacity that may be added. In such a case theaggregate with added temporary capacity becomes a new donor aggregatewith respect to other aggregates that are entitled to have addedcapacity. It remains in the state of donor until all the temporary addedcapacity is removed.

-   -   Added bandwidth is assigned using a combination of donor        aggregates and the donor re-assignable unused bandwidth so that        the choice of donors is based on the amount of available        capacity they have to donate including, but not limited to,        selecting the first donor as the one with the largest amount of        capacity available to donate and taking all this capacity and        then, if necessary, selecting the donor with the next largest        capacity and continuing in this way until the limit of what can        be added to this aggregate has been reached, where this limit on        the total added capacity is specific to each aggregate.

Function 6.3.1.2 indicates to 6.3.3 and 6.3.2 any change in the rates orthresholds that should now be used as a result of a change of capacity,relating to that aggregate and the donor aggregates. Function 6.3.1.2maintains the remaining unused capacity of any donor re-assignableunused capacity.

-   -   Thresholds will include any rate thresholds that should be        increased or decreased based on the change of capacity    -   Thresholds will include any rate jitter thresholds that should        be changed, based on the change of capacity.

FIG. 9 shows another embodiment where arriving signaling packets arerecognized by the classifier function 6.1 and are passed to function6.5. In turn, function 6.5 can inform function 6.3 of signals receivedthat relate to an external Verification Function. Similarly, function6.3.1.1 will indicate to function 6.5 the requirement to pass on anindication to an external function 6.3.1.2 using signaling packets thatare passed back to function 6.1 for onward transmission.

FIG. 10 illustrates an exemplary embodiment of a computer platform uponwhich the inventive system may be implemented.

FIG. 10 is a block diagram that illustrates an embodiment of acomputer/server system 1000 upon which an embodiment of the inventivemethodology may be implemented. The system 1000 includes acomputer/server platform 1001, peripheral devices 1002 and networkresources 1003.

The computer platform 1001 may include a data bus 1005 or othercommunication mechanism for communicating information across and amongvarious parts of the computer platform 1001, and a processor 1005coupled with bus 1001 for processing information and performing othercomputational and control tasks. Computer platform 1001 also includes avolatile storage 1006, such as a random access memory (RAM) or otherdynamic storage device, coupled to bus 1005 for storing variousinformation as well as instructions to be executed by processor 1005.The volatile storage 1006 also may be used for storing temporaryvariables or other intermediate information during execution ofinstructions by processor 1005. Computer platform 1001 may furtherinclude a read only memory (ROM or EPROM) 1007 or other static storagedevice coupled to bus 1005 for storing static information andinstructions for processor 1005, such as basic input-output system(BIOS), as well as various system configuration parameters. A persistentstorage device 1008, such as a magnetic disk, optical disk, orsolid-state flash memory device is provided and coupled to bus 1001 forstoring information and instructions.

Computer platform 1001 may be coupled via bus 1005 to a display 1009,such as a cathode ray tube (CRT), plasma display, or a liquid crystaldisplay (LCD), for displaying information to a system administrator oruser of the computer platform 1001. An input device 1010, includingalphanumeric and other keys, is coupled to bus 1001 for communicatinginformation and command selections to processor 1005. Another type ofuser input device is cursor control device 1011, such as a mouse, atrackball, or cursor direction keys for communicating directioninformation and command selections to processor 1005 and for controllingcursor movement on display 1009. This input device typically has twodegrees of freedom in two axes, a first axis (e.g., x) and a second axis(e.g., y), that allows the device to specify positions in a plane.

An external storage device 1012 may be coupled to the computer platform1001 via bus 1005 to provide an extra or removable storage capacity forthe computer platform 1001. In an embodiment of the computer system1000, the external removable storage device 1012 may be used tofacilitate exchange of data with other computer systems.

The invention is related to the use of computer system 1000 forimplementing the techniques described herein. In an embodiment, theinventive system may reside on a machine such as computer platform 1001.According to one embodiment of the invention, the techniques describedherein are performed by computer system 1000 in response to processor1005 executing one or more sequences of one or more instructionscontained in the volatile memory 1006. Such instructions may be readinto volatile memory 1006 from another computer-readable medium, such aspersistent storage device 1008. Execution of the sequences ofinstructions contained in the volatile memory 1006 causes processor 1005to perform the process steps described herein. In alternativeembodiments, hard-wired circuitry may be used in place of or incombination with software instructions to implement the invention. Thus,embodiments of the invention are not limited to any specific combinationof hardware circuitry and software.

The term “computer-readable medium” as used herein refers to any mediumthat participates in providing instructions to processor 1005 forexecution. The computer-readable medium is just one example of amachine-readable medium, which may carry instructions for implementingany of the methods and/or techniques described herein. Such a medium maytake many forms, including but not limited to, non-volatile media andvolatile media. Non-volatile media includes, for example, optical ormagnetic disks, such as storage device 1008. Volatile media includesdynamic memory, such as volatile storage 1006.

Common forms of computer-readable media include, for example, a floppydisk, a flexible disk, hard disk, magnetic tape, or any other magneticmedium, a CD-ROM, any other optical medium, punchcards, papertape, anyother physical medium with patterns of holes, a RAM, a PROM, an EPROM, aFLASH-EPROM, a flash drive, a memory card, any other memory chip orcartridge, or any other medium from which a computer can read.

Various forms of computer readable media may be involved in carrying oneor more sequences of one or more instructions to processor 1005 forexecution. For example, the instructions may initially be carried on amagnetic disk from a remote computer. Alternatively, a remote computercan load the instructions into its dynamic memory and send theinstructions over a telephone line using a modem. A modem local tocomputer system can receive the data on the telephone line and use aninfrared transmitter to convert the data to an infra-red signal. Aninfra-red detector can receive the data carried in the infra-red signaland appropriate circuitry can place the data on the data bus 1005. Thebus 1005 carries the data to the volatile storage 1006, from whichprocessor 1005 retrieves and executes the instructions. The instructionsreceived by the volatile memory 1006 may optionally be stored onpersistent storage device 1008 either before or after execution byprocessor 1005. The instructions may also be downloaded into thecomputer platform 1001 via Internet using a variety of network datacommunication protocols well known in the art.

The computer platform 1001 also includes a communication interface, suchas network interface card 1013 coupled to the data bus 1005.Communication interface 1013 provides a two-way data communicationcoupling to a network link 1015 that is coupled to a local network 1015.For example, communication interface 1013 may be an integrated servicesdigital network (ISDN) card or a modem to provide a data communicationconnection to a corresponding type of telephone line. As anotherexample, communication interface 1013 may be a local area networkinterface card (LAN NIC) to provide a data communication connection to acompatible LAN. Wireless links, such as well-known 802.11a, 802.11b,802.11g and Bluetooth may also be used for network implementation. Inany such implementation, communication interface 1013 sends and receiveselectrical, electromagnetic or optical signals that carry digital datastreams representing various types of information.

Network link 1013 typically provides data communication through one ormore networks to other network resources. For example, network link 1015may provide a connection through local network 1015 to a host computer1016, or a network storage/server 1017. Additionally or alternatively,the network link 1013 may connect through gateway/firewall 1017 to thewide-area or global network 1018, such as an Internet. Thus, thecomputer platform 1001 can access network resources located anywhere onthe Internet 1018, such as a remote network storage/server 1019. On theother hand, the computer platform 1001 may also be accessed by clientslocated anywhere on the local area network 1015 and/or the Internet1018. The network clients 1020 and 1021 may themselves be implementedbased on the computer platform similar to the platform 1001.

Local network 1015 and the Internet 1018 both use electrical,electromagnetic or optical signals that carry digital data streams. Thesignals through the various networks and the signals on network link1015 and through communication interface 1013, which carry the digitaldata to and from computer platform 1001, are exemplary forms of carrierwaves transporting the information.

Computer platform 1001 can send messages and receive data, includingprogram code, through the variety of network(s) including Internet 1018and LAN 1015, network link 1015 and communication interface 1013. In theInternet example, when the system 1001 acts as a network server, itmight transmit a requested code or data for an application programrunning on client(s) 1020 and/or 1021 through Internet 1018,gateway/firewall 1017, local area network 1015 and communicationinterface 1013. Similarly, it may receive code from other networkresources.

The received code may be executed by processor 1005 as it is received,and/or stored in persistent or volatile storage devices 1008 and 1006,respectively, or other non-volatile storage for later execution.

Finally, it should be understood that processes and techniques describedherein are not inherently related to any particular apparatus and may beimplemented by any suitable combination of components. Further, varioustypes of general purpose devices may be used in accordance with theteachings described herein. It may also prove advantageous to constructspecialized apparatus to perform the method steps described herein. Thepresent invention has been described in relation to particular examples,which are intended in all respects to be illustrative rather thanrestrictive. Those skilled in the art will appreciate that manydifferent combinations of hardware, software, and firmware will besuitable for practicing the present invention. For example, thedescribed software may be implemented in a wide variety of programmingor scripting languages, such as Assembler, C/C++, perl, shell, PHP,Java, etc.

Moreover, other implementations of the invention will be apparent tothose skilled in the art from consideration of the specification andpractice of the invention disclosed herein. Various aspects and/orcomponents of the described embodiments may be used singly or in anycombination in the communications network and methods of operating of acommunications network. It is intended that the specification andexamples be considered as exemplary only, with a true scope and spiritof the invention being indicated by the following claims.

What is claimed is:
 1. A method of operating a packet subnetworkcomprising: partitioning of one or more physical links into a number ofaggregates, where each aggregate is assigned an amount of capacity andat least one aggregate is assigned to one or more donor aggregatesco-existing on the same link or links as the assigned aggregates, sothat any one of the at least one aggregate may increase its capacity byborrowing capacity from one or more of its donor aggregates, or maydecrease its capacity by giving back capacity to one or more of thedonor aggregates; receiving one or more communication packets indicativeof a commencement of a packet communication through a subnet, the packetcommunication comprising a flow consisting of a plurality of packets;responsive to recognition of the commencement of a new packet flow,storing a flow identifier, enabling identification of packets belongingto the commenced packet communication flow and associating that flowidentifier with one or more of the aggregates, including or excludingthe donor aggregates, retrieving policy information relating to anyinitial or sustained QoS treatment of the flow, and determining aninitial or sustained discard probability; and, responsive to a policydetermination and a QoS treatment determination related to the flowidentifier, retrieving the policy determination and QoS treatmentdetermination associated with the flow of a latest arriving packet, anddetermining if one or several predetermined thresholds or limits havebeen reached, or will be reached if the latest arriving packet istransmitted.
 2. The method of claim 1, further comprising, responsive todetermining that one or more predetermined thresholds or limits havebeen reached on the aggregate associated with the flow of the latestarriving packet: causing an indication to be observable that an increaseor decrease in capacity is required for that aggregate and acorresponding decrease or increase of capacity is required of one ormore donor aggregates associated with that aggregate; and enablingimmediate transfer of the latest arriving packet subject only to thedetermination of the discard probability associated with the flowidentity of the latest arriving packet.
 3. The method of claim 1,wherein the policy information relating to any initial or sustained QoStreatment of the flow is derived from at least one of: packet headerinformation at any level of encapsulation; signaling information relatedto that flow identity; policy attributes associated with the physicallink on which the packet arrived or will depart; measured rates applyingto a flow rate of the flow, or a measured jitter in flow rate; andmeasured rates applying to the total traffic on one or more of theaggregates which the new flow is associated with, or the measured jitterin such rates.
 4. The method of claim 1, wherein the predeterminedthresholds are based on at least one of: flow rate or jitter in ratemeasured on any flow; flow rate or jitter in flow rate measured on anyaggregate; or flow count in any aggregates.
 5. The method of claim 1,wherein the predetermined thresholds are based on at least one of: flowdiscard probability on the flow of the latest arriving packet; packetdiscard measurements on any aggregate; or packet discard measurements onany flow.
 6. The method of claim 1, wherein the predetermined thresholdsare based on at least one of: received signaling related to any flow, orreceived signaling related to any aggregate.
 7. The method of claim 1,wherein the predetermined thresholds are based on policy information onany aggregates associated with the flow of the latest arriving packet.8. The method of claim 1, wherein the predetermined thresholds are basedon packet delay measurements on any flow or averaged on any aggregate.9. The method of claim 1, wherein the predetermined limits or thresholdsindicative of a change in bandwidth requirement on one or moreaggregates associated with the flow identity of the latest arrivingpacket comprise measurements of rates or jitter in rates associated witha group of flows which may or may not include the flow of the latestarriving packet, and where this group of flows is a subset of at leasttwo active flows of one or more of the aggregates associated with theflow identity of the latest arriving packet.
 10. The method of claim 1,wherein the predetermined limits or thresholds indicative of a change inbandwidth requirement on one or more aggregates associated with the flowidentity of the latest arriving packet comprise a preference priorityassociated with the flow identity of the latest arriving packet, derivedby a policy determination and a QoS treatment determination or through asignaling received about the flow.
 11. The method of claim 1, wherein adonor aggregate has no flows and its capacity is an unused pool ofcapacity usable to increase the capacity of other aggregates and itscapacity is increased by the release of capacity from other aggregates.12. The method of claim 1, wherein the observable indication that achange in aggregate capacity is required on one or more aggregatesincludes the sending of an indication towards a verification functionthat determines if the indicated change in capacity can be permitted ormodified.
 13. The method of claim 12, wherein the verification functionoptionally reduces the capacity of a donor aggregate by sending anindication towards a policing function that lowers one or more policingthresholds on rate or rate jitter of the donor aggregate such that, whenthe lowered policing thresholds are determined to be exceeded bymeasurement of the latest and recent arriving packets of the totaltraffic of the donor aggregate, a number of packets are discardedinstead of being transmitted so that the remaining transmitted packetsdo not exceed the lowered policing rates.
 14. The method of claim 12,wherein the verification function, having observed the indication for anincrease in capacity on an aggregate, optionally determines for thisaggregate which donor aggregates associated with the aggregate shall bedecreased in capacity and by what amount of capacity.
 15. The method ofclaim 12, wherein at least one unused pool of capacity is shared amongseveral other flow aggregates and the verification function controls thesharing of its pool of capacity such that the verification functionallows temporary use of any spare capacity in the unused capacity poolfor any aggregate indicating that an increase in capacity is required onat least one basis of temporary or unguaranteed assignments above apredetermined share of the pool of capacity.
 16. The method of claim 12,wherein at least one unused pool of capacity is shared among severalother flow aggregates and the verification function controls the sharingof its pool of capacity such that the verification function decreasesthe temporary use of spare capacity when other aggregates require anincrease in capacity and one or more aggregates are currently using morethan a predetermined share of the unused pool of capacity, such that noaggregate is decreased in capacity below its predetermined share unlessit indicates that a further decrease is required, wherein a donoraggregate has no flows and its capacity is an unused pool of capacityusable to increase the capacity of other aggregates and its capacity isincreased by a release of capacity from other aggregates.
 17. The methodof claim 15, wherein the determination of an initial or sustaineddiscard probability associated with a flow identity includes theassignment of a larger discard probability when, for any aggregateassociated with the flow identity, being an aggregate that has a portionof its capacity assigned on at least one basis of the temporaryassignment of capacity, a measured rate of that aggregate is greaterthan one or more threshold rates based on the predetermined share ofcapacity available to that aggregate.
 18. The method of claim 12,wherein an unused pool of capacity is shared among several other flowaggregates and the verification function controls the sharing of itspool of capacity such that: the verification function allows guaranteedassignments of additional capacity for any aggregate indicating that anincrease in capacity is required on the basis of guaranteed assignmentsup to a predetermined share of the pool of capacity; and theverification function decreases capacity of any aggregate requiringguaranteed assignments of capacity.
 19. The method of claim 17, wherein,when a packet arrives whose flow is associated with an aggregate, wherecongestion has ceased or where traffic levels on an aggregate havefallen below some threshold indicative of the onset of congestion, thenretrieving policy relating to such a change of status of this aggregateand relating to re-grading of the discard probability of the flow, giventhat measured levels have fallen or congestion has disappeared.
 20. Themethod of claim 17, wherein, when measurements relating to anyrate-based, byte count-based, or packet-count based threshold shows thatmeasured levels are now below level(s) that had previously exceeded oneor more such thresholds, then retrieving policy relating to such achange of status of this aggregate and relating to re-grading of thediscard probability of the flow, given that measured levels have fallenor congestion has disappeared.